20 Gaussian Process

Recall bivariate normal distribution: ${\displaystyle (Y_1,Y_2)\sim {\mathcal{N}}_2(\overrightarrow{\mu },\mathrm{\Sigma })}$, where $$\overrightarrow{\mu }=\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\end{array}\right),\mathrm{\Sigma }=\left(\begin{array}{cc}{\sigma }_1^2& \rho {\sigma }_1{\sigma }_2\\ \rho {\sigma }_1{\sigma }_2& {\sigma }_2^2\end{array}\right).$$
We are curious about ${\displaystyle Y_2|Y_1=a\sim \mathcal{N}({\mu }_{2|1},{\mathrm{\Sigma }}_{2|1})}$. See Lecture 19, this implies $${\mu }_{2|1}={\mu }_2+\rho \frac{{\sigma }_2}{{\sigma }_1}(a-{\mu }_1),{\mathrm{\Sigma }}_{2|1}={\sigma }_2^2(1-{\rho }^2)\le {\sigma }_2^2.$$
How can we visualize ${\displaystyle Y_1,\cdots ,Y_d}$ for ${\displaystyle d>2}$?
Gaussian Process generalizes this concept to functions ${\displaystyle Y\{x\in \mathbb{R},x\in S\subset \mathbb{R}\}}$, ${\displaystyle S}$ can contain infinitely many points.

• Mean function: can be anything. Popular choices: ${\displaystyle m(x)=0,\mathrm{\forall }x;m(x)={\beta }^{\mathrm{T}}x}$.
• Covariance function:
  • Symmetric
  • PSD (positive semi-definitive)
  • Stationary: ${\displaystyle \mathrm{Cov}(Y(x),Y(x^{\prime }))=k(x,x^{\prime })=k(x-x^{\prime })}$.
  • Isotropic: ${\displaystyle \mathrm{Cov}(Y(x),Y(x^{\prime }))=k(||x-x^{\prime }||)}$, is a stronger condition. ${\displaystyle ||\cdot ||}$ is an arbitrary norm.

• Positive definite.
• GP is Infinitely differentiable.

Sampling procedure:

1. Discretize ${\displaystyle S}$ as ${\displaystyle \{x_1,\cdots ,x_D\}}$.
2. Sample ${\displaystyle Y(x_1)\sim \mathcal{N}(m(x_1),k(x_1,x_1))}$.
3. For ${\displaystyle n=1,\cdots ,D-1}$, ${\displaystyle (Y(x_1),\cdots ,Y(x_{n+1}))\sim {\mathcal{N}}_{n+1}(\overrightarrow{m},K)}$, where $$\overrightarrow{m}=\left[\begin{array}{c}m(x_1)\\ ⋮\\ m(x_n)\\ m(x_{n+1})\end{array}\right],K=\left[\begin{array}{cccc}k(x_1,x_1)& \cdots & k(x_1,x_n)& k(x_1,x_{n+1})\\ ⋮& \ddots & ⋮& ⋮\\ k(x_n,x_1)& \cdots & k(x_n,x_n)& k(x_n,x_{n+1})\\ k(x_{n+1},x_1)& \cdots & k(x_{n+1},x_n)& k(x_{n+1},x_{n+1})\end{array}\right].$$
   Denote ${\displaystyle {\overrightarrow{C}}_{n+1}=[k(x_1,x_{n+1})\cdots k{(}_n,x_{n+1}){]}^{\mathrm{T}}}$.
   Sample ${\displaystyle Y(x_{n+1})}$ from ${\displaystyle Y(x_{n+1})\mid Y(x_1),\cdots ,Y(x_n)\sim \mathcal{N}({\mu }_{n+1},{\sigma }_{n+1}^2)}$. So $$\begin{align*}
   \mu_{n+1}&=m(x_{n+1})+\vec{C}{n+1}^{\mathrm{T}}K{n}^{-1}\begin{bmatrix}
   Y(x_{1})-m(x_{1}) \ \vdots \ Y(x_{n})-m(x_{n})
   \end{bmatrix},\
   \sigma_{n+1}^{2}&= k(x_{n+1},x_{n+1})-\vec{C}{n+1}^{\mathrm{T}}K{n}^{-1}\vec{C}_{n+1}.
   \end{align*}

$$$$

• ! ${\displaystyle K_n}$ can be ill conditioned, leading to numerical issues.
  A solution: replace ${\displaystyle K}$ with ${\displaystyle K+{\tau }^{2}I}$, where ${\displaystyle {\tau }^{2}I}$ is a constant diagonal matrix. This is similar to the regularization in Ridge regression.
  Interpretation: noisy observation. Each ${\displaystyle Y(x_i)}$ is observed with some additive noise independent of GP: ${\displaystyle Y(x_i)=g(x_i)+{\epsilon }_i}$. ${\displaystyle {\epsilon }_i\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }N(0,{\tau }^2)}$ are called the nugget, which are independent of GP.

Common kernels:

• RBF Kernel: $$K_{\mathrm{RBF}}(x,x^{\prime })={\sigma }^2\exp \{-\frac{1}{2l^2}|x-x^{\prime }{|}^2\}.$$

• Matern Kernel: ${\displaystyle \nu =p+\frac{1}{2}}$ where ${\displaystyle p\in \mathbb{N}}$. $$K_{p+1/2}^{\mathrm{Matern}}(x,x^{\prime })=\frac{{\sigma }^{2}p!}{(2p)!}\sum _{j=0}^p\frac{(p+j)!}{j!(p-j)!}{[\frac{2\sqrt{2p+1}}{l}|x-x^{\prime }|]}^{p-j}\exp (-\frac{\sqrt{2p+1}}{l}|x-x^{\prime }|).$$ The summation is a polynomial in ${\displaystyle |x-x^{\prime }|}$ of degree ${\displaystyle p}$. So GP with ${\displaystyle K_{\nu }^{\mathrm{matern}}}$ is ${\displaystyle \lceil \nu \rceil -1}$ times differentiable in the "mean square" sense.

• Brownian Motion (Non-stationary): $$K_{\mathrm{BM}}(x,x^{\prime })=min(x,x^{\prime }),x,x^{\prime }\in (0,\mathrm{\infty }).$$

• Brownian Bridge (Non-stationary): $$K_{\mathrm{BB}}(x,x^{\prime })=min(x,x^{\prime })-x{x}^{\prime },x,x^{\prime }\in [0,1].$$
  This corresponds to a stochastic process conditioned on ${\displaystyle Y(0)=0,Y(1)=0}$.

• Ornstein-Uhlenbeck: $$K_{\mathrm{OU}}(x,x^{\prime })=\frac{{\sigma }^{2}l}{2}\exp \{-\frac{1}{l}|x-x^{\prime }|\},x,x^{\prime }\in (0,+\mathrm{\infty }).$$ This model has applications in physics, biology, finance, etc.

• Linear (Non-stationary): Bayesian Linear Regression. Suppose ${\displaystyle Y(x_i)={\beta }_0+{\beta }_1(x_i-c)}$, where ${\displaystyle {\beta }_0\sim N(0,{\sigma }_0^2),{\beta }_1\sim N(0,{\sigma }_1^2)}$ and ${\displaystyle {\beta }_0\perp \perp {\beta }_1}$. Then for ${\displaystyle x\ne x^{\prime }}$, $$\begin{array}{rl}\mathrm{Cov}(Y(x),Y(x^{\prime }))=& \mathrm{Cov}({\beta }_0+{\beta }_1(x-c),{\beta }_0+{\beta }_1(x^{\prime }-c))\\ =& {\sigma }_0^2+{\sigma }_1^2(x-c)(x^{\prime }-c).\end{array}$$ So the corresponding kernel is $$K_{\mathrm{linear}}(x,x^{\prime })={\sigma }_0^2+{\sigma }_1^2(x-c)(x^{\prime }-c).$$

• Prediction:
  Training data ${\displaystyle D=\{(x_i,y_i),i=1,\cdots ,n\}}$. How can we predict ${\displaystyle Y(x)}$ for a new sample point ${\displaystyle x}$? We should use sampling procedure.